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important areas of concern about the Air Force ROTC teaching and courses. 
Findings suggest that the previous evaluation instrument contained two 
questions unrelated to stakeholder concern and lacked questions related to 
six areas of concern. It was not considered to be an effective method of 
evaluating teaching and course concerns. Thirteen appendixes contain 
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While the Air Force ROTC (AFROTC) Detachment at a large northeastern university 
received an “Excellent” rating on a recent inspection, the staff continued to feel there was room 
for continuous improvement as the highest rating is “Outstanding.” Before the inspection, a new 
tool, the mid-semester student evaluation of teaching, was designed and implemented; however, 
it only received a brief evaluation by the education officer, which lacked formal validation. 
Therefore, the problem addressed in this research project was that the faculty of the detachment 
was unable to effectively alter teaching deficiencies before the end of a semester. 

The purpose of this study was to determine if the previous questions within the mid- 
semester student evaluation of teaching were the most valuable for the students and faculty. 
There was one significant research question involved with this study. “Is the current student 
evaluation of teaching an effective measure of teaching quality as perceived by both students and 
faculty?” 




An interview instrument and protocol were developed based on an extensive literature 
review. Formative and summative committees were utilized consisting of experienced and 
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unbiased academics external to the Air Force department. These committees provided 
continuous feedback during the study’s interview development stage. The interview instrument 
was pilot tested and implemented for use with faculty and students participating in AFROTC 
education at the university. 

The interviews conducted resulted in the compilation of important areas of concern, as 
noted by the faculty and students, about the AFROTC teaching and courses. When the raw data 
was compared to the current literatiire, these areas became evident and quantifiable. At the 
conclusion of data analysis it was determined that the previous evaluation system contained two 
questions unrelated to stakeholder concern and lacked six areas of concern, thus resulting in an 
ineffective method of evaluating teaching and course concerns. 

The desire for feedback is an important aspect in the performance of any employee. This 
is potentially more important when the job involves other people instead of inanimate products. 
Both students and teachers retain various opinions on what is important in the range of 
educational feedback. Due to the variability in opinions, it is important to include input from all 
stakeholders. Beyond the need for stakeholder involvement, is the need to continuously evaluate 
results. No program is perfect, thus leaving room for improvement after implementation. 

The assessed program of teaching evaluation discussed will positively influence the 
detachment at the university and should result in improved instructor confidence and student 
development. It was recommended that the current teaching evaluation system receives 
necessary changes, a follow-on evaluation, continuous improvement, and is shared with other 
AFROTC detachments. 
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Chapter 1 

INTRODUCTION 

The United States Air Force mandates the Air Force Reserve Officer Training Corps 
(AFROTC) program, located at over 142 colleges and universities, with the task of providing 
qualified officer accession. The AFROTC program is provided to students who must complete a 
variety of courses before becoming an Air Force officer. Active duty Air Force officers, from 
diverse operational assignments in an array of career fields, are assigned to teach and oversee 
these courses. Examples of the instructors’ career fields include pilot, engineer, aircraft or 
missile maintenance, and intelligence. While the active duty officers assigned to these teaching 
positions have a wide range of experiences, they often have little or no experience as teachers. 

Nature of the Problem 

The AFROTC detachment at a large northeastern four-year university found success in its 
current operations through the attainment of an “excellent” rating on an Operational Readiness 
Inspection conducted by the Air Force’s Air Education and Training Command. The education 
program, individually, received the same rating; however, there was room for improvement as 
the highest rating is “outstanding.” A new tool, the mid-semester student evaluation of teaching, 
was designed and implemented (Appendix A); however, it only received a brief evaluation by 
the education officer, which lacked formal validation. The education officer was unable to state 
that the program was valid based on the needs of the detachment faculty and students. 

Therefore, the problem was that the faculty of the detachment was unable to effectively alter 
teaching deficiencies before the end of a semester. 
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Purpose of the Study 

The purpose of this study was to determine if the previous questions within the mid- 
semester student evaluation of teaching were the most valuable for the students and faculty. This 
study helped determine the effectiveness of the previous program utilized by the detachment and 
its ability to provide the faculty with functional information. 

Significance to the Institution 

The majority of professors in other academic areas at the university hold a Doctoral 
Degree; however, instructors assigned to AFROTC do not hold similar credentials. This fact 
places AFROTC faculty at a distinct professional disadvantage as it relates to comparative 
instructional effectiveness. As part of a traditional Doctoral program, candidates often spend 
many hours teaching classes under the tutelage of an experienced faculty member, thereby 
accumulating practical experience. In an effort to maintain a similar level of academic aptitude, 
AFROTC must provide various instruments of quality assessment so that their students perceive 
the instructors as the academic equivalent of the university’s professors. While a system of 
measuring effectiveness was in place, it had not been validated through rigorous research. 
Therefore, this researcher believed that the current system must be validated in an effort to 
ensure quality instruction takes place in the AFROTC programs at the university. 

Research Question 

This study answered the following research question: “Is the current student evaluation 
of teaching an effective measure of teaching quality as perceived by both students and faculty?” 
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Definition of Terms 

For the purpose of this study, the following terms need further clarification. 

Assigrunent. This term refers to the particular job a member of the Air Force is currently doing 
or going to in the future. 

Cadet. This title refers to a college student seeking an Air Force commission through the ROTC 
program. A cadet is legally considered a college student and not covered by the Uniform Code 
of Military Justice. 

Detachment. The Air Force uses this term as the organizational name for a ROTC imit at a 
college or university. A detachment serves its host school plus a number of other institutions. 
Education Officer. The AFROTC faculty member who is responsible for continual staff 
development and evaluation. 

Officer. This is a commissioned military member appointed by the President of the United 
States. 

Operational. This is the imofficial designation used to denote a imit, within the Air Force, that 
performs duties directly related to the mission of the Air Force. Examples are aircraft and 
nuclear missile imits. 

ROTC. The Reserve Officer Training Corps is a program designed by the Department of 
Defense to produce commissioned officers for the various military services of the United States. 
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Chapter 2 

REVffiW OF LITERATURE 
Introduction 

Often the word “evaluation” rings a note of concern across a teacher’s ear. This concern 
may result from the fear of criticism, the loss of a coveted tenure position, or even removal from 
a job. Fortunately, this is the evaluation of the past. Recent uses of faculty evaluations, 
especially in the higher education setting, have focused on teacher development (Mayo, 1997, p. 
1). Instead of forming a file of teacher criticism, the evaluation process now forms a veritable 
textbook of teaching education. This literature review takes a deeper look into the realm of 
student evaluations of teaching. Initially, the review presents evidence from both faculty and 
student points of view. Secondly, the review briefly presents information on specific areas of 
teaching effectiveness. Finally, the review will present information concerning the needs 
assessment and the interview method of evaluation. 

Evaluation Concerns 

Viewing the vast dynamics of the classroom environment, it is not difficult to see that 
two predominate elements include the teacher and the student. Due to the nature of this 
interaction between two people, one must endeavor to seek an understanding from both 
perspectives. 

Faculty Concerns 

Accountability is the name of the game when it comes to performance. This single word 
has been a driving factor in evaluating faculty at the end of each term, which often is the 
dominant method of quantifying teaching competency (Gullatt & Ballard, 1998, p. 4; Hightower, 
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1998, p. 1; Mertler, 1999, p. 4). The vast body of research showed varying levels of faculty 
concern and support for the use of these evaluations to determine teaching quality. 

Hightower (1998) found that student evaluators may not view their comments as 
seriously as faculty or administrators would like. This is a special concern when an instructor’s 
career may be at risk (p. 2). This lack of seriousness is evidenced in the following student 
statement: “The TA steadily improved throughout the course. . .1 think he started drinking, and it 
really loosened him up” (Hightower, 1998, p. 2). In conjunction with the factor of seriousness 
was a concern about whether students are being honest in their remarks. Mertler (1999) found 
teachers were accepting of student feedback but questioned how honestly students were actually 
commenting. The teachers felt an evaluation would not be useful unless the students were 
making honest comments (p. 8). However, to a greater degree than student-related factors, a lack 
of communication between faculty and administrators provided the largest contribution to faculty 
concern (Gullatt & Ballard, 1998, p. 1). 

Gullatt & Ballard (1998) found a relation between faculty acceptance of student 
evaluations and the support received from administrators (p. 5). They noted that the 
improvement of teaching, and not the evaluation of faculty performance, should be the driving 
theme behind teaching evaluations (p. 1). Andrews (1997) considered the use of Total Quality 
Management (TQM) techniques in the evaluation of teaching. TQM is a business-oriented 
theory that focuses on continuous improvement and management at all levels (p. 1). Recently, 
many colleges have attempted to introduce these values into the evaluation of faculty. As with 
many new theories, there was some dissention among the faculty. Concerns included the ability 
to view the student as a customer, interference with teaching expertise, and the threat to 
academic freedom (p. 2). Though these represented real concerns, many administrations found 
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ways to overcome them, to the benefit of the faculty. In-service training and faculty involvement 
in the TQM process were popular methods. Another important factor was the method in which 
administrators work with the faculty. The administration must listen effectively, provide 
constructive feedback, and engage in joint goal setting with the faculty (p. 3). 

Through communication and involvement, many faculty members became excited about 
the use of student evaluations of teaching and looked forward to the results (Mertler, 1999, p. 7). 
Mertler found very positive reactions from teachers he used in a study of student evaluations of 
teaching with all teachers providing ratings of “extremely” or “somewhat useful” (p. 8). In 
addition to the usefulness, he found appropriateness rated equally high with 83% rating 
“extremely appropriate,” and 17% rating the evaluation “somewhat appropriate” (p. 8). The 
same teachers, at a 75% rate, stated they found evaluation use as feasible and 64% would use the 
survey twice per academic year (p. 9). Mertler found the key to the evaluation success was the 
opporhmity for the faculty to provide input on the instrument. “It is crucial to note that. . .they 
were given opporhmities to review the draft of the SE3T [title of instrument] instrument for 
purposes of providing suggestions or revisions. . .” (Mertler, 1999, p. 5). Brackbill (1996) 
supports this finding in his implementation of teaching evaluations. His success came when the 
teachers were assured that only teachers would see the evaluation results. This established an 
environment of quality improvement over performance evaluation (p. 1). 

Student Concerns 

Research consistently showed that students are very interested in what they learn, how 
they learn it, and from whom they learn it. Students are no longer passive listeners in the 
classroom; they are becoming active partners with knowledgeable faculty (Osborne, 1998, p. 4). 
“Essentially, the good teacher tells us what is out there to learn, shows an enthusiasm for 
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acquiring knowledge. . .and then turns us loose to learn at our own pace. . .” (Belton, 1996, p. 1). 
Belton recounted that students want their teacher to bring a subject to life through active learning 
as opposed to drab readings from a textbook (p. 1). In concert with the prevailing research, 
Belton called for the use of students to evaluate teaching effectiveness (p. 2). 

Research studies among the literature supported the varying nature of effectiveness 
factors, as well as the ability to measure teaching effectiveness (Jirovec, Ramanathan & Alvarez, 
1998, p. 1; Osborne, 1998, p. 4; Sheehan and Duprey, 1999, p. 3). In their study of psychology 
students, at both the graduate and imdergraduate levels, Sheehan and Duprey (1999) used a 27- 
item instrument and found five items strongly reflecting students’ desires. These items included 
lecture information, assessment tools, instructor preparation, interesting lecture format, and 
challenging material (p. 3). An interesting result, though, was that students rated their desire to 
take another class from the instructors lower than all other items, even though the students rated 
the instructors as highly effective (p. 4). Similar to the Sheehan and Duprey study, Jirovec, et al. 
(1998) performed a correlation study on social work student response to teaching evaluations. 
They also foimd strong correlations with teaching effectiveness and student ratings on 
organization, grading, and rapport for students (p. 4). Interestingly, though, is the finding that 
students preferred elective courses (considered more popular) over required courses for the major 
(p. 4). This finding was slight, however, and the more concrete variables of organization, 
grading and rapport were stronger predictors of effectiveness (p. 5). “The key predictor of 
student ratings was organizational skill of the instructor. . .. Students are telling us that an 
organized approach to teaching contributes to better education” (Jirovec, et al., 1998, pp. 5-6). 

The classroom should be a learning community that is open and capable of change. Clear 
feedback can allow instructors to expand their awareness and understanding, and to grow and 
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change. The feedback loop in the wider learning community allows for a continuous flow of 
information that can be incorporated by the instructor to improve and change the system. 
(Osborne, 1998, p. 2). 

Teaching Effectiveness 

Consistent with research among faculty and students, the literature identified specific 
characteristics that are indicative of an effective instructor. Examples of such characteristics 
included class organization and student interaction (Jackson, Teal, Raines, Nansel, Force, & 
Burdsal, 1999, pp. 7-8; Shannon & Twale, 1998, p. 446; Young & Shaw, 1999, p. 678). 
Instructors who display these, and similar characteristics, were often viewed as successful and 
effective teachers. The body of literature reviewed for this research helped answer important 
questions relevant to narrowing the most appropriate teaching characteristics. Questions such as 
“What defines an effective instructor?” and “What are the characteristics of an instructor defined 
as effective?” were reviewed. 

In an effort to determine the characteristics of an effective instructor, many researchers 
have utilized student surveys. These characteristics included learning or course value, concern 
for students, course organization, fairness, difficulty of course, and workload (Jackson, et al., 
1999, pp. 7-8; Shannon & Twale, 1998, p. 446; Young & Shaw, 1999, p. 678). This research 
showed a consistent pattern of responses from students, which define an instructor’s 
effectiveness. Each of the above researchers also found additional characteristics not consistent 
among their studies, which included motivating students, effective communication, group 
interaction, enthusiasm, and breadth of coverage (Shannon & Twale, 1998, p. 446; Young & 
Shaw, 1999, p. 678). By using these characteristics, it is believed that a topically knowledgeable 
person could develop into an effective instructor. A question to consider though is “Must all the 
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characteristics be present?” Some researchers think not, for example, Young and Shaw (1999) 

found that instructors could have certain deficiencies; nevertheless, they still receive high ratings 

fi"om students. They posited that an effective teacher is one who can compensate for weak areas 

by demonstrating outstanding skills in other vital areas of teaching (p. 683). 

The drama of classroom life is a multilayered, multifaceted fabric of experiences that, to 
the untrained outsider, must seem like a crazy quilt.... Answers reduce anxieties and 
restore a sense of inner security. Classroom problems elevate teachers’ anxieties. 

Having to decide what to do, choosing fi-om many possible courses of action, is a high- 
risk endeavor, full of potential hazards. This is the stuff that keeps teachers awake nights 
and, when they finally sleep, makes for disturbing dreams. (Wassermann, 1999, p. 2) 

The intense nature of such a multifaceted environment may manifest itself in the ways 
instructors feel about themselves and their ability to teach. Therefore, confidence in one’s 
abilities is an important factor when considering teaching effectiveness. Fritz and Miller-Heyl 
(1995) found this to be true in their study of teacher self-efficacy. That is, an instructor who is 
confident in his or her abilities will create a cycle of self-motivation (p. 1). “Teaching efficacy 
leads to a greater willingness to try new methods, which may result in higher achievement in 
students, which, in turn, increases a teacher’s sense of efficacy and willingness to continue to 
expand greater efforts to help students learn” (Fritz & Miller-Heyl, 1995, p. 2). In essence, a 
willingness to try novel techniques is perceived as favorable by students and their positive 
feedback motivates the instructor to try new methods. They found significant increases in the 
confidence level and self-satisfaction in teachers receiving training compared to a non-trained 
group, which showed significant decline (p. 7). This level of confidence goes beyond the 
teacher’s subject matter competence and suggests that teachers must know how to teach their 
content area and subject matter. 
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Needs Assessment 

While the above literature presented relevant information to the use of student 
evaluations of teaching, it is also important to review techniques for determining their ability to 
meet the needs of the students and faculty who utilize those evaluations. According to Varcoe 
(1994) the needs assessment is a tool useful to determine the difference between what a program 
provides and what its constituents need (p. 9). “It can be effective in determining specific needs 
and using them to focus development of approaches for inclusion in the design (or redesign) of 
programs” (Varcoe, 1994, p. 9). 

Kemp, Morrison and Ross (1998) support the views of Varcoe and find the needs 
assessment as an effective tool in both identifying a problem and choosing a solution (p. 21). In 
essence, when a problem exists, important needs of individual or groups are no longer met. 

These needs are seen as the difference between what the current situation is and how the 
organization’s constituents feel how the situation should be (p. 21). Kemp, et al. viewed these 
needs from six various definitions. Normative needs are those based on some form of data based 
on national statistics (p. 21). If the organization’s needs are compared to some external group, 
the needs are considered comparative (p. 22). When needs are not compared to some known 
factor, yet still evident in an individual the need is considered felt (p. 23). When those felt needs 
are stated in some means they are considered expressed (p.23). If needs may occur in the future, 
yet are not currently felt or expressed, they are considered anticipated (p. 24). Finally, an 
emergent situation within an organization that produces a gap is considered a critical incident 
need (p. 24). 

Champion (2000) provides further support to the existence of needs and the use of a 
needs assessment for determining gaps. He views the assessment tool as an opportunity to not 
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only scan the environment for existing problems, but also as a tool to obtain vital information 

from those involved in the current needs gap (p. 1). 

One of the commandments of professional development is: Thou shalt not plan programs 
in a vacuum. . ..Adult learners want to be involved in making decisions about their 
learning and their involvement has been proven over and over to enrich the results 
(Champion, 2000, p. 1). 

Champion further noted that the conduct of a needs assessment is an important aspect in its 
success. Designers must overcome the perception that needs assessments are formal and 
mmidane and pursue a new approach where the process is informal and meets multiple needs (p. 
1). An effective tool in meeting this need can be fornid in the face-to-face interview (Champion, 
2000, p. 2; Kemp, et al., 1998, p. 23). 

Interview Methodology 

The interview method of data collection allows the researcher to obtain the specific needs 
necessary for the assessment. In addition, it allows the researcher to provide important 
clarification or seek further details, when needed, during the data collection stage (NSU, 1994, p. 
4). Kemp, et al. (1998) posited that an interview method allows the respondents to express their 
needs in a low anxiety environment (p. 23). “Felt needs are best expressed through interviews 
and questionnaires. Face-to-face interviews are often more effective, since the designer can 
alleviate anxieties and probe for additional details” (Kemp, et al., 1998, p. 23). Unfortunately, 
interviews do possess various disadvantages, which may include high cost, increased time and 
effort, and lack of availability of necessary respondents (NSU, 1994, p. 4). However, if 
performed correctly, the interview methodology can produce important results. 
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Design 

The design of the interview is a delicate procedure resulting in an instrument that will 
retrieve qualitative information from individuals. LeCompte (2000) stated that an important 
aspect of the interview instrument is the lack of any bias (p. 1). People, as human begins, tend to 
approach their feeling with a sense of ownership regarding various situations. They may answer 
questions based solely on the topic (p. 1). Therefore, the researcher must understand the 
instrument and the data desired from the respondent when performing a needs assessment 
through interviews (King, 1997, p. 1). “The questions form a script for us to use, but like every 
good actor, we should know our lines well before the curtain rises” (Dilley, 2000, p. 3). 

In designing the instrument. King (1997) considers various elements as important to the 
success of the data collection. The interviewer should not only intimately imderstand the 
elements of the questions but also reserve the ability to alter them for understanding. In addition, 
design the interview with an imderstanding of what questions may lead to potential bias. 
Furthermore, the instrument must be imderstood by the interviewer to provide confidence to the 
respondents in the interviewer’s knowledge (p. 2). While King places emphasis on the 
interviewer, Sorenson (1999) goes one step further and focuses on the respondents. His 
predominate concern was that the instrument be designed with the respondent in mind (p. 2). 
“Our interview designs must acknowledge that our field teams are facing the real live consumer, 
not a paper ideal” (Sorenson, 1999, p. 2). 

Dilley (2000) provides further support to designing the interview around various 
respondents. Gathering information about the topics surroimding the interview as well as 
information about the respondents is an important aspect of design (p. 1). This includes a 
research of literature surroimding the area of concern. Understanding the discipline in which the 
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interview resides provides important information on what questions to ask and how to ask them 
(p. 2). “The questions we ask should create a bridge between person and report, between content 
and form as well as (background) information and (personal) experience” (Dilley, 2000, p. 2). 

Conduct 

The body of current research points to many factors in conducting a successful interview. 
King (1997) posited that interviewers must consider carefully how the interview is conducted. In 
the beginning, the interviewer should spend time to create a relationship with the respondent. 

The questions should then be asked as they are on the instrument with various probes for detail. 
However, the interviewer must avoid biasing the sessions by providing possible answers (p. 1). 

Dilley (2000) supports the notions set forth by King and provides further support to the 
concern of the overall relationship created. Dilley’ s research stipulates that the respondent feel 
comfortable during the interview, thus allowing the interviewer to access deeper feelings (p. 3). 

In order to accomplish this, the interviewer must establish the role of the listener. Through 
intense listening, the interviewer can often determine the true meanings behind what a 
respondent is feeling (p. 6). “One rule of thumb is for interviewers to talk 20 percent of the time 
during the interview, and listen 80 percent” (Dilley, 2000, p. 5). 

In addition to establishing rapport, listening offers the interviewer the ability to alter the 
course of the interview if necessary. The interviewer must compare what the respondent states, 
to what the interviewer knows from previous knowledge. This will allow the interviewer to 
make qualitative decisions on when to alter the course of the interview protocol (Dilley, 2000, p. 
4; King, 1997, pp. 2-3). This alteration is considered probing, where the interviewer asks further 
questions based on the responses given. King (1997) states that the correct way to probe during 
an interview is to ask questions that clarify responses or insure that perceptions are correct (p. 3). 
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Probing questions by the interviewer should not provide possible answers, opinions, or guide the 
respondent’s answers in any way (p. 3). Often these actions lead to a biasing factor within an 
interview session (p. 3). 

The purpose of the interview is for the interviewer to seek necessary information from a 
respondent. Unfortunately, there is an opportunity for bias to enter the session. When the 
interviewer provides specific answer possibilities, or gives their personal opinion about the 
questions posed, the interview becomes biased. In this case, the interviewer is using his or her 
knowledge to guide the respondent in a pre-determined direction, which is often that desired by 
the researcher (King, 1997, pp. 2-3). Avoiding these behaviors as well as providing adequate 
response time will lead to a more valid interview session (p. 3). 

Analysis 

The use of an interview methodology allows the researcher to obtain various 
informational pieces from respondents, thereby resulting in qualitative information. This 
qualitative information provides a rich compilation of experiences and feelings, which the 
researcher can utilize to determine patterns of belief or behavior (Dilley, 2000, p. 2). To find 
these patterns the researcher must choose an appropriate population as the test sample. Once the 
sample is chosen, and the interview provided, the researcher must then analyze the data. 

Various methods are available for determining the appropriate sample to analyze, 
including both random and non-random. When utilizing random samples, a probable sample 
provides all members of the population an equal opportunity to participate while a systematic 
sample chooses participants based on a particular counting number, such as every 10th person. 

A further random method is stratified, in which a certain population of participants is randomly 
chosen based on their percentage of the total population (NSU, 1994, p. 9). Non-random choices 
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include convenience and quota samples and lack the random nature of the previous methods. 
Convenience samples are based on the researcher’s ability to utilize a particular sample, such as 
all co-workers or students in a particular setting. Quota samples are similar to the stratified 
methodology, however the population itself is chosen based on convenience and the percentage 
drawn from that group (p. 10). When dealing with needs assessments, the non-random sample 
may provide the most benefit to the researcher, thus the sample size may be small. However, 
Kemp, et al. (1998) noted that the sample size should be chosen based on the desired population. 
Therefore, a needs assessment of a small population can be just as significant as a larger 
population provided the sample is representative of those possessing the needs (p. 27). 

Once the sample is chosen and the data collected, the researcher must analyze all 
information, thereby determining the results (LeCompte, 2000, p. 1). LeCompte considers 
analysis as the building of pieces of information into a logical conclusion. Smaller pieces are 
combined into larger ideas, which are linked into end results (p. 2). Analysis is accomplished in 
five steps, tidying up, finding items, creating sets, creating patterns, and assembling structures 
(pp. 3-6). 

After all data is collected, the researcher must compile all the information into a logical 
system. This may include cataloging, labeling, or indexing in an effort to make the data easier to 
review (LeCompte, 2000, p. 3). Secondly, the researcher finds the items of interest in support of 
the research. This may include frequency of responses, responses omitted by the respondents, or 
significant item declared directly by the respondents during the data collection (pp. 3-4). Once 
data items are determined, the third step, creating items, is utilized to place the data into large 
groups for comparison and subdivided as necessary (pp. 3-4). These items may include 
demographic data, ranking, opinions, or feedback from the respondents (NSU, 1994, p. 6). “The 
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purpose of these activities is to clump together items that are similar or go together” (LeCompte, 

2000, p. 4). The various items may then be analyzed together for data patterns relevant to the 

research (p. 5). Finally, the patterns are combined to create the final descriptive structure of the 

data, thereby offering evidence to support or refute the research questions (p. 6). 

Analysis that is meticulously done, based on clearly articulated theories, and responsive 
to research questions can be good analysis. However, to create good research findings, 
analysis must also yield results that are meaningful to the people for whom they are 
intended and described in a language they understand. (LeCompte, 2000, p. 7) 

Summary 

The above literature review brought three main points to the forefront of this study. First, 
the review showed that both faculty and students desire the use of student evaluations of teaching 
as a method of improving classroom quality. However, faculty members often have concerns 
over how seriously the students take the evaluations. Secondly, the review demonstrated that 
identifiable measurements of teaching effectiveness exist and teachers who understand them are 
more successful in the classroom. Finally, the review presented evidence that the interview 
method of evaluation is an appropriate instrument to ensirre valid results in a needs assessment. 
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Chapter 3 

METHODOLOGY AND PROCEDURES 
Data Collection and Analysis 

As a beginning step to the evaluation process, a review of current theoretical knowledge 
was completed. This review focused on current theories relating to student evaluations of 
teaching as well as faculty and student perceptions of those instruments. Furthermore, literature 
surrounding the needs assessment as an evaluation tool was reviewed. Finally, the researcher 
conducted a review of literahare relating to structured interviews as a system of data collection. 

Second, after a thorough review of relevant research was completed, criteria for success 
were established. A review of current theories, and the use of a formative committee, aided in 
selecting criteria. The formative committee provided initial and continuing direction throughout 
the design phase of the interview instrument. This committee consisted of an Enrollment 
Counselor at a community college and a Director of Consumer Relationship Management at a 
large Health Management Organization. The researcher chaired the formative committee 
(Appendix B). This committee of experts provided consistent oversight of the interview design 
process and they reviewed initial design, proposed changes, reviewed drafts, and provided other 
necessary feedback concerning the interview instrument. An informational letter was sent to 
committee members upon approval of the project proposal (Appendix C). 

Third, an initial draft of the interview instrument was designed, which consisted of all 
appropriate questions as determined necessary by the formative committee. The formative 
committee reviewed the draft to ensure its consistency with the stated criteria. Following this 
approval the final draft was created for submission to the summative committee. The formative 
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committee members were presented with an appreciation gift, consisting of a printed card and a 
holiday candle in ajar, valued at $3.00. 

Fourth, validation of the interview instrument and its criteria were performed by the use 
of a summative committee. This committee consisted of an Associate Professor of Naval 
Science at a private university, and the Assistant Academic Dean at a religion-affiliated college. 
The researcher chaired the summative committee (Appendix B). This committee reviewed the 
interview instrument to ensure it met the previously determined criteria. Feedback from the 
summative committee was compiled and implemented, resulting in a final design for the 
interview instrument. An informational letter was sent to all committee members upon approval 
of the proposal (Appendix C). The summative committee members were presented with an 
appreciation gift, consisting of a printed card and a holiday candle in ajar, valued at $3.00. 

Fifth, the interview instrument was pilot tested, after validation by the summative 
committee. The researcher provided the interview to one AFROTC staff member, one under- 
class cadet, and one upper-class cadet. The selection of respondents was accomplished using a 
quota basis, selecting one officer staff member, the first under-class cadet, and first upper-class 
cadet present in the cadet lounge when the researcher entered the lounge. This provided a 
similar composition to the actual interview pool (NSU, 1994, p. 10). Before beginning the 
interview questions, the researcher explained the purpose of the study and the instrument, and 
provided an informed consent letter (Appendix D) for each to review and sign. During the 
interview, the researcher recorded the respondents’ answers to the questions as well as their 
statements about the quality of the interview instrument and session (p. 28). This was 
accomplished utilizing both written notes and a tape recorder. Upon completion, the researcher 
offered each respondent a candy bar valued at $1 .00. 
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Sixth, the validated interview instrument was implemented as the means of evaluating the 
current mid-semester student evaluation of teaching. The interview pool consisted of the 
instructors of the freshmen, sophomore, and senior AFROTC courses, and two cadets from each 
of the freshmen through senior AFROTC courses. The junior-year instructor was not 
interviewed, as he is the researcher. There is only one instructor for each academic year; 
therefore, they were chosen by convenience and not randomly. However, the cadet respondents 
were chosen randomly through a volunteer process. The researcher asked the Cadet Commander 
(cadet in charge of all other cadets) to request volunteers from each year group. He then chose 
two cadets from each year group with the researcher absent during the selection process. 
Throughout each interview session, the purpose of the session was explained and a consent form 
was signed (Appendix D). The researcher posed the questions as prescribed in the instrument’s 
directions. In addition, the researcher took hand-written notes and utilized a tape recorder during 
the interview for analysis purposes. Upon completion, the researcher offered each respondent a 
candy bar valued at no more than $1.00. 

Seventh, the researcher, upon completion of all interviews, tabulated the interview 
responses. After reading the handwritten notes and comparing them to the audio tape recordings, 
the researcher compiled the responses by question and respondent type (Appendix L). The types 
of responses were then organized into key concepts, based on literatiure and researcher 
determination, and scored based on their frequency of mention by the respondents (Appendix 
M). The frequencies were then compared to frequency averages for all questions in their 
respective categories (Instructor Related or Course Related). This comparison allowed the 
analysis of high or low need areas. A simple average was computed for each category. Instructor 
Related and Course Related, which quantifies the average number of response made per area of 
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concern. For the areas registering concern (question 1-2 and 4-5), response rates were 
considered high-importance (greater than 3 above average), average-importance (-3 to +3 of the 
average) and low-importance (less than 3 below average) above the average. For the areas 
registering no concern (questions 3 and 6) response rates below the average were representative 
of areas of unimportance to the respondents. 

Eighth, as the final step to the evaluation process, the data was compared to the current 
mid-semester student evaluation of teaching (Appendix A), to determine if it met the determined 
needs. A final report was completed, documenting the needs of the faculty and students. 

Assumptions 

It was initially assumed that the formative and summative committees chosen were 
capable of providing objective and knowledgeable feedback to the interview design process, and 
measurement of validity based on the established criteria. It was also assumed that current 
theories of student evaluations of teaching were the most current and useful for AFROTC faculty 
members. Finally, it was assumed that the interview respondents provided honest answers to the 
valid interview questions and did not attempt to purposefully deceive the interviewer. 

Limitations 

The final product is limited to use only by the AFROTC Detachment at the university of 
study. The student evaluation of teaching validated may not be sufficiently general to allow its 
use in traditional academic departments. Future regulatory changes by the Air Force or the 
university may alter the evaluation results. 
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Chapter 4 
RESULTS 

To evaluate the current mid-semester student evaluation of teaching program, an eight- 
step process was utilized. The first step included a thorough review of the related literature. 

This review revealed evidence for the use of such teaching evaluations to support the needs of 
both students and faculty. While both students and faculty entertain concerns about the use of 
teaching evaluation instruments, the general notion is that their benefits far outweigh any 
disadvantages (Belton, 1996, p. 2; Hightower, 1998, p. 2; Mertler, 1999, pp. 7-8; Osborne, 1998, 
p. 4). The literature also supported the notion that teaching effectiveness is a measurable aspect 
and distinct qualities are evident among effective teachers (Jackson, et al., 1999, pp. 7-8; 

Shannon & Twale, 1998, p. 446; Young & Shaw, 1999, p. 678). Use of these instruments allows 
students to express their needs as well as providing teachers with an opportunity to improve their 
skill. The logical conclusion of this observation is that both students and faculty possess various 
needs within the teaching environment and those needs are measurable. Therefore, a researcher 
may utilize a needs assessment to determine the effectiveness of a designed teaching evaluation 
instrument (Kemp, et al. 1998, p. 21). In doing so, the interview methodology is likely the best 
methods for determining the needs exhibited by various constituents in the learning process 
(Kemp, et al., 1998, p. 23; NSU, 1994, p. 4). Based on this literature, the researcher chose the 
interview methodology to evaluate the effectiveness of the current mid-semester student 
evaluation of teaching utilized by the AFROTC Detachment at the university. 

The second step of the methodology involved the formation of a committee to provide 
guidance and direction in creating the interview instrument. Initial feedback on criteria for 
creating a successful instrument was similar among committee members. The researcher 
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obtained feedback using an email questionnaire (Appendix E). The universal themes included 
the level of comfort and confidence between interviewer and respondent, as well as concerns 
surrounding potential biasing factors. The committee received an initial draft of criteria based on 
their input and the literature review (Appendix F). The committee approved the criteria with no 
changes (Appendix G). 

The third step of the methodology involved the creation of the interview instrument. 
Referring to the criteria, the interview questions, as well as a protocol of conduct were created. 
The initial draft of the interview instrument was sent to the formative committee for their review, 
with their responses tabulated on a standardized feedback form (Appendixes H, I, & J). A 
concern surfaced over the analysis of the instrument, and the use of a tape recorder was added to 
the use of hand-written notes for data collection. 

The fourth step of the methodology began the validation stage of the project. The final 
draft, as determined by the formative committee was submitted to the summative committee for 
their review and validation with their responses tabulated on a standardized feedback form 
(Appendixes I, J, & K). The committee validated the instrument stating that it met all criteria 
established through, and approved by the formative committee. One committee member 
emphasized the confidentiality of respondent answers, since a department faculty member 
(researcher) was used to conduct the interview. However, he felt that the questions were benign 
enough to elicit fair and honest answers fi'om both student and faculty respondents. The other 
member elicited concern over the need for a blind study; however, this was not considered a 
concern among the current literature and was therefore not added to the protocol. 

The fifth step of the methodology consisted of a pilot test of the designed and committee- 
validated instrument. This step allowed the researcher to determine that the instrument met all 
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criteria established and validated by the committees (Appendix G). Meeting the first design 
criterion, the interview was conducted face-to-face with only the interviewer and respondent 
present. Evidence supporting the second design criterion was found in that all respondents stated 
they felt the instrument allowed them to express their needs as they relate to the classroom 
environment. Meeting the final design criterion, all respondents stated that they never felt biased 
toward a particular answer. While the researcher had to provide clarification on certain 
questions, the respondents stated they were never provided information that could be construed 
as a response example. The first conduct criterion was met in that all respondents stated they felt 
comfortable during the interview and believed that their responses would be kept in confidence. 
Each respondent scheduled his or her own time to meet with the interviewer, thereby meeting the 
second design criterion, convenience. Finally, the questions were initially read to each 
respondent in the same manner, however the interviewer reworded questions at the respondents 
request, thereby meeting the third conduct criterion. The pilot test resulted in no changes to the 
instrument; therefore, the researcher considered the instrument a valid measure of student and 
faculty needs. 

The interview instrument was implemented during the sixth step. Interviews were 
conducted at the AFROTC Detachment on the university’s campus. The interviews occurred 
over the period of 1-8 December 2000. After volunteering for the interviews, all respondents 
provided a time they could meet with the researcher at their convenience. At the interview 
session, all respondents signed an informed consent letter (Appendix D) and were given an 
opportunity to ask any general questions about the study. The researcher reiterated the 
confidentiality of their responses and recorded all responses through handwritten notes and a 
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tape recorder. The interview was conducted according to directions set forth in the interview 
instrument (Appendix I) and respondents answered all questions posed by the researcher. 

During the seventh step of the methodology, the researcher reviewed the interview 
responses and tabulated the data based on frequency of mention (Appendix M). The questions 
were broken into two significant categories. Instructor Related (questions 1-3) and Course 
Related (questions 4-6). Data in these two categories were reviewed to determine major areas of 
concern (LeCompte, 2000, pp. 3-6). The major areas of concern were drawn from literatme 
when possible and area additions or deletions were made based on researcher review. Areas 
drawn from literatme included: Course Organization, Effective Communication, Group/Student 
Interaction, Instructor Interest/Enthusiasm, Rapport/ Availability for Students, Methods of 
Presentation/Leaming, Course Value, Interesting Material/Information, Breadth of Coverage, 
and Workload (Jackson, et al., 1999, pp. 7-8; Jirovec, et al., 1998, p. 4; Shannon & Twale, 1998, 
p. 446; Sheehan and Duprey, 1999, p. 3; Young & Shaw, 1999, p. 678). Areas created by the 
researcher based on respondent data included: Visual Aids, Instructor Knowledge, Instructor 
Physical Characteristics, Teaching Skill, and Other Students. 

The review of tabulated responses produced significant areas of concern from the 
respondents. In the category of Instructor Related Questions, the areas of Course Organization, 
Visual Aids, Instructor Knowledge, and Instructor Physical Characteristics produced little 
concern from the respondents. The areas of Effective Communication, Group/Student 
Interaction, and Instructor Interest/Enthusiasm reflected average concern. The areas of greatest 
concern were Rapport/Availability, and Methods of Presentation/Leaming. The low frequency 
of responses, to the areas of no concern, supported the notion that students and faculty are 
concerned about instructor characteristics. Furthermore, the areas of average and high concern 
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all produced low or no frequencies in the areas registering no concern. The only exception was 
the area of Methods of Presentation/Leaming, which received six responses showing no concern 
for this area; however, this still represented a low concern when compared to the average 
response rate of 18 . 

In the category of Course Related Questions, the areas of Breadth of Coverage, 
Workload, and Instructor Physical Characteristics represented low concern. The areas of 
Teaching Skill, Course Value, Interesting Material/Information, Learning Environment, and 
Other Students represented average concern. The area of Course Organization was the only area 
to register high concern. The low frequency of responses, to the areas of no concern, supported 
the notion that students and faculty are concerned about course characteristics. Furthermore, the 
areas of average and high concern all produced low or no frequencies in the areas registering no 
concern. The exceptions were Course Organization and Learning Environment, which received 
frequencies of four and eight respectively. 

As the eighth and final step to the methodology, the data was compared to the current 
mid-semester student evaluation of teaching (Appendix A) and a final report was completed to 
document the needs of the faculty and students. Questions one and two of the current evaluation 
represented good questions, as they respectively evaluate Group/Student Interaction and 
Rapport/ Availability for Students. Question three was considered a poor question as it evaluates 
Instructor Knowledge, which did not emerge as a pressing need for students or faculty. 

Questions four and five were goods questions as they evaluate Effective Communication and 
Course Value, both areas of average concern. Question six registered as a poor question, as 
Visual Aids did not produce great concern from the respondents. Questions seven and eight 
registered as good questions as they evaluate Course Organization, an area of high concern for 
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course related needs. While six of eight questions evaluated areas of concern, the current 
evaluation instrument was missing other areas. These included Methods of Presentation/ 
Learning, Instructor Interest/Enthusiasm, Teaching Skill, Interesting Material/Information, 
Learning Environment, and Other Students. 

There was one research question considered for this project. “Is the current student 
evaluation of teaching an effective measure of teaching quality as perceived by both students and 
faculty?” Based on a validated interview instrument, both students and faculty were interviewed 
to determine their needs. A great amount of information was accumulated from all the 
respondents, who spoke candidly and freely to the researcher. As a result, multiple areas of 
concern were determined for both instructor and course related categories. Upon extensive 
review of the data, the researcher determined that while the current mid-semester student 
evaluation of teaching represents a good foundation of assessment, it lacks vital areas of concern. 
In addition, it contains some questions, which do not produce desired information for either 
faculty or student populations. Overall, the instrument was determined to be a good foundation, 
but an ineffective measure of student and faculty needs. 
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Discussion 



The results of this study support the current literature reviewed by the researcher. The 
inclusion of both faculty and student input proved successful and resulted in changes to the 
previous survey. Andrews (1997) found the inclusion of all parties important in his review of 
Total Quality Management techniques in academic environments (p. 1). Since the researcher is 
part of the faculty, the study provided students and fellow faculty an opportunity to share their 
concerns with the detachment staff (p. 2). This tenet is extremely important in alleviating the 
faculty concerns when implementing policy changes (p. 2). 

The inclusion of both faculty and staff resulted in input similar to studies of student 
evaluations of teaching. Both students and instructors felt that course value, concern for 
students, and course organization were important factors in concluding a course or instructor is 
effective (Jackson, et al., 1999, pp. 7-8; Shannon & Twale, 1998, p. 446; Young & Shaw, 1999, 
p. 678). These factors were found important by all the studies above. Interview results also 
support the findings found in two of the three studies above. These include effective 
communication, group interaction, and enthusiasm (Shannon & Twale, 1998, p. 446; Young & 
Shaw, 1999, p. 678). 

Wasserman (1999) stated that the classroom environment is a multifaceted environment 
that causes great anxiety for teachers (p. 2). This anxiety may result fi'om lack of knowledge and 
feedback about what is effective. It is no wonder this study supported Rothwell (1996) in his 
conclusion that feedback is an important factor in the success of performance (p. 245). To 
alleviate the anxiety of teaching, instructors must receive consistent, timely, and informative 
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feedback. Once instructors are confident in their teaching abilities, they become more willing to 
take educational risks and create a more effective and interesting learning environment (Fritz & 
Miller-Heyl, 1995, p. 1). The important aspect here is the need for informative feedback. 
Students and teachers must operate together, sharing information resulting in total quality and 
total involvement (Andrews, 1997, p. 3). 

Just as students and instructors retain opinions about effective teaching abilities and 
environments, this study supported the notion that they are just as willing to share those 
opinions, thereby expressing their needs. To accomplish this, a needs assessment returns results, 
which are both quantifiable and useful to an evaluator (Kemp, et al., 1998, p. 21 ; Varcoe, 1994, 
p. 9). 

Both the formative and summative committees supported the use of a face-to-face 
interview to determine felt needs expressed by the stakeholders (Champion, 2000, p. 2; Kemp, et 
al., 1998, p. 23). The criteria designed with the formative committee and validated by the 
summative committee, were similar to the prevailing literature. Therefore, an interview designed 
as faice-to- face (Kemp, et al., 1998, p. 23), seeking pertinent information (Dilley, 2000, pp. 1-2), 
and lacking bias (LeCompte, 2000, p. 1) was a paramount concern to the committees and 
researcher during this study. Furthermore, the conduct of the interview session displayed 
considerable attention by the literature and committees. The focus here resulted in criteria based 
on the environment and convenience (Dilley, 2000, p. 3) and the use of probing questions 
(Dilley, 2000, p. 4 ; King, 1997, pp. 2-3). 

The process of research, which resulted in a valid interview instrument and data, strongly 
supported the literature at large. Input from the formative committee and validation by the 
summative committee added additional corroboration that the face-to-face interview process 
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perpetuates the accumulation of important data concerning effective learning among students. 
This is further substantiated when those students and faculty are participants in affecting that 
learning environment. Any evaluation system used to measure teaching and course effectiveness 
must endeavor to meet the variable needs of students who have a desire learn and the faculty 
who desire to teach them. To accomplish this feat, though, systems of evaluation must be built 
upon a fluid frame, which contorts to a copious populace, and is consistent with the ideals of 
continuous improvement (Osborne, 1998, p. 2). 

Conclusions 

In a world where compromises in quality lead to disastrous consequences in results, 
knowledge is a primary factor in success. This is found to be true on factory floors, department 
store shelves, and in university lectures. Customers demand quality, and research indicates they 
are willing to pay for it. Moreover, when they do not receive quality service or products, they 
either voice their dissatisfaction or leave their current supplier (Sellers, 1998, p. 200). In the 
educational community, universities are the providers of a product, namely education, and their 
customers are the students who pay for this product, in hope of obtaining skills, knowledge, and 
abilities that will improve their chances of obtaining a successful career. They pay great sums of 
money to obtain the knowledge that will guarantee a successful life. With such a prominent 
demand by the customer, why would anyone consider allowing an ineffective instructor to 
conduct an ineffective course that its customers, the students, paid for? This desire for quality is 
often the impetus for evaluating instructors as well as the courses they teach. 

The desire for feedback is an important aspect in the performance of any employee. This 
is potentially more important when the job involves other people instead of inanimate products. 
Both students and teachers retain various opinions on what is important in the range of 
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educational feedback. Due to the variability in opinions, it is important to include input from all 
stakeholders. This ensures complete needs coverage for groups who differ in opinion. Often the 
greatest need is information. Not all instructors need continual training involving many hours 
away from work. Many just seek effective and frequent feedback from students and peers. 

Beyond the need for stakeholder involvement, is the need to continuously evaluate 
program results. No program is perfect, thereby leaving room for improvement from the point of 
implementation. As departments create and implement new programs, evaluation should take a 
place at the top of the priority list, especially in the enviromnent where teachers may have never 
taught before. Their anxieties are high enough, and without adequate feedback, those anxieties 
may spin out of control. One cannot underestimate the importance of providing teachers with 
adequate feedback from students who want input in their learning enviromnent. Students 
demand better teachers, and teachers naturally want to provide the best for their students. 



The assessed program of teaching evaluation discussed above will positively influence 
the detachment at the university and should result in improved instructor confidence and student 
development. A corrected mid-semester student evaluation of teaching, which is based on valid 
interview data and supported by academic research, will provide an effective method for 
determining the quality of instruction provided to the AFROTC cadets. This program meets the 
needs of both students and faculty, and the program’s implications have the potential to reach far 
beyond meeting the evaluation needs of only the studied university’s AFROTC detachment. 

The use of an effective evaluation of teaching, based on the needs of instructors and 
students, will positively affect the success of the AFROTC Detachment at the university. The 
survey provides feedback to the instructors at the mid-point of each semester and the information 
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received includes aspects of both teaching technique and course success. Since the questions are 
built from student and instructor input, both major stakeholders will find opportunities to excel. 
Instructors will receive input on topics important to those receiving the instruction and the 
students are provided an opportunity to alter their educational environment. This may also foster 
closer educational relationships between student and instructor. 

Outside the university’s AFROTC Detachment, the potential for success extends to other 
AFROTC detachments as well. Through information sharing on the WWW, other detachments 
may learn how to create and utilize effective evaluations of teaching based on the needs of all 
stakeholders. This not only increases the quality of education at the studied university, but at 
hundreds of other universities nationwide. 

Recommendations 

It was recommended that the current mid- semester student evaluation of teaching receive 
the following changes. Remove questions three and six concerning Instructor Knowledge and 
Visual Aids respectively. In addition, the detachment should add questions concerning 
Presentation and Teaching Skill, Course Value, Material Interest, Instructor Enthusiasm, Other 
Students, and the Learning Environment. This will result in a net gain of four questions; 
however, they will provide a more effective measure of student and faculty need. 

It was recommended that the corrected student evaluation of teaching receive a follow-on 
evaluation as soon as feasible. This evaluation is the responsibility for the education officer who 
consolidates all student evaluations of teaching. By reviewing past end-semester evaluations 
with the upcoming spring 2001 evaluations, a conclusion on the success of this evaluation study 
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Another recommendation was that the detachment ensures the student evaluation is 
continuously updated, possibly on an annual basis, to keep pace with changing educational 
theory and stakeholder needs. To accomplish this, the education officer should continue to 
review current literature and informally question both students and faculty. In conjunction with 
the feedback evaluation, annual program review will help maintain a high level of effectiveness 
and validity. 

Finally, it was recommended that the detachment share this new information with other 
detachments around the nation. Beyond sharing the corrected evaluation form, the detachment 
should share the methods used in obtaining the questions. This will allow other detachments to 
successfully design evaluations based on the needs of their instructors and students. 
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APPENDIXES 




Appendix A 

Current Mid-Semester Student Evaluation of Teaching 



Aerospace Studies Academic Course 
Mid-Semester Evaluation of Teaching 

The purpose of this evaluation is to let your instructor know how you feel the class is progressing at the semester 
mid-point. This form is anonymous, so be honest. Please answer the questions below and turn this form into your 
class instructor. Thank you for helping make your class better. 

Please use the following scale in rating this course. 



Strongly Disagree 


Disagree 


Neither Agree 
or Disagree 


Agree 


Strongly Agree 


1 


2 


3 


4 


5 



1 . The instructor seeks active participation from students 



1 


2 


3 


4 


5 



2. The instructor is actively helpful when students have problems 



1 


2 


3 


4 


5 



3. The instructor is knowledgeable in the subject matter 



1 


2 


3 


4 


5 



4. The instructor speaks audibly and clearly 



1 


2 


3 


4 


5 


5. The instructor stimulates my thinking 


1 


2 


3 


4 


5 


6. The instructor effectively uses appropriate audio/visual and technology resources 


1 


2 


3 


4 


5 


7. The instructor organizes the class well 


1 


2 


3 


4 


5 


8. The course material (texts, handouts, etc.) enhance quality instruction 


1 


2 


3 


4 


5 



What do you like best about your AS Class so far? 



What do you like least about your AS Class so far? 
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Appendix B 

Description of Committee Members 



Formative 



Enrollment Counselor 



Qualifications : This committee member is an instructor and counselor at a Community College. 
He currently provides admissions, enrollment processing and academic counseling for the 
college’s business division, as well as high school and area business recruiting. He holds a 
Masters Degree in Education and Post-Masters work in Diploma School Counseling. 

Purpose for selection : This individual currently teaches and advises undergraduate students, and 
has a direct understanding of what both instructors and students desire. His current studies in 
higher education bring the knowledge of research methods into the committee. His experience 
and training brings knowledgeable value to the evaluation process. 

Method of selection : Telephone call to discuss project, and provision of proposal. 

Director of Consumer Relationship Management 

Qualifications : This committee member has extensive experience in the field of training and 
education. She has held many position of management in the areas of adult education and 
curriculum development. Her current work consistently requires her expert development and 
evaluation skills. She holds a Masters Degree in Teaching and Curriculum and a Doctorate in 
Adult Education. 

Purpose for selection : This individual has held many academic and corporate positions requiring 
education evaluation skills. Her experience and training brings knowledgeable value to the 
evaluation process. 

Method of selection : Telephone call to discuss project, and provision of proposal. 
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Summative 



Associate Professor, Naval Science 



Qualifications : This committee member is a Navy officer holding the rank of Commander. He 
is currently the Executive Officer for a Naval ROTC detachment at a private university. His 
years of military service predominately consist of time as a Naval Flight Officer flying the F-14 
fighter aircraft. He has also held a variety of positions in persoimel, training, and aircraft 
maintenance. He holds a Masters Degree in Education Curriculum and Instruction. 

Purpose for selection : This committee member is knowledgeable of the military training 
enviromnent as well as education theory. He brings both internal understanding as a military 
member and an external component holding a position outside the researcher’s department. 

Method of selection : Telephone call to discuss project, and provision of proposal. 

Assistant Academic Dean 



Qualifications : This committee member is an Assistant Dean involved with program 
development, assessment, curriculum and course offerings. In addition, he sits on the college’s 
curriculum committee. He holds a Masters Degree in Theatre, and is currently pursuing a 
Doctorate in Higher Education. 

Purpose for selection : This member was chosen due to his administrative role in higher 
education. He was chosen for his informed, but unbiased input potential. 

Method of selection : Telephone call to discuss project, and provision of proposal. 
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Letter to Committee Members 
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1 Dec 99 

MEMORANDUM FOR COMMITTEE MEMBERS 
FROM: AFROTC DET/EO 



SUBJECT: Study Committee Information 

1. First off, thank you for volunteering to assist me with my current academic project. I 
appreciate the offer of your time and expertise during this part of my Doctoral program at Nova 
Southeastern University. Attached to this letter you will find a copy of my proposal explaining 
the details of my project. Please take a few moments to review it, as it will help you understand 
the reasons behind this project. 

2. As part of my methodology, there will be two evaluative committees, formative and 
summative, both chaired by myself Each will convene approximately three times, through e- 
mail, for the purpose of reviewing the progress of my project. In these sessions, you will each 
provide valuable feedback concerning criteria, format, and content. All contact will be through 
email in an effort to reduce your time requirements. Pertinent information on committee 
members is as follows: 

Name Committee 

Enrollment Counselor Formative 

Director of Consumer Relationship Management Formative 
Assistant Academic Dean Summative 

Associate Professor of Naval Science Summative 



3. Once again, thank you for your help and guidance. I will be in touch soon. If you have 
further questions concerning this issue, contact me. 



//SIGNED// 

KEVIN C. SELLERS, Capt, USAF 
Assistant Professor, Aerospace Studies 
Education Officer 

Attachment: 

Proposal 



O 

ERIC 
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Appendix D 

Letter of Informed Consent 



MEMORANDUM FOR RESPONDENTS 
FROM: AFROTCDET/EO 



SUBJECT: Informed Consent Letter 

Title: AN EVALUATION OF THE MID-SEMESTER STUDENT EVALUATION OF 
TEACHING AT A UNIVERSITY AIR FORCE RESERVE OFFICER TRAINING CORPS 
DETACHMENT 

Investigator: 

Kevin C. Sellers, Doctoral Student, Programs for Higher Education, Nova Southeastern 
University. 

Institutional Review Board, Office of Grants and Contracts, Nova Southeastern University, (954) 
262-5369 

Description of the Study: The purpose of this study is to determine the most effective questions 
to utilize on the mid-semester student evaluations of teaching. Through the assistance of a 
formative and summative committee, the researcher will design an interview survey. The survey 
will be provided to faculty and student respondents to determine their needs as they relate to the 
current program. Greater detail of the study may be found in the proposal. 

Risks and Benefits to the Participant: There are no expected risks to respondents. 

Costs and Payments to the Participant: There is no expected cost to respondents. Committee 
members will not receive payment for services to this project other than a small gift of 
appreciation not to exceed $5. Interview respondents will not receive payment for services to 
this project other than a candy gift of appreciation. 
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Confidentiality: Strict confidentiality will be maintained at all times throughout the course of the 
research project unless law requires disclosure. Committee office phone numbers, addresses, 
and email will be shared with the researcher and other committee members. Appropriate Nova 
Southeastern University faculty will also view them during evaluation of the report. Interview 
respondent information will not be shared. 

Participant's Right to Withdraw from the Study: You may choose to not participate or to stop 
participation in the research program at any time without penalty. 

Voluntary Consent by Participant: Participation in this research project is totally voluntary, and 
your consent is required before you can participate in the research program. 



//SIGNED// 

KEVIN C. SELLERS, Capt, USAF 
Assistant Professor, Aerospace Studies 
Education Officer 



I have read the preceding consent form, or it has been read to me, and I fully understand 
the contents of this document and voluntarily consent to participate. All of my questions 
concerning the research have been answered. I hereby agree to participate in this research 
study. If I have any questions in the future about this study the investigator listed above 
will answer them. A copy of this form has been given to me. 

Participant's Signature: Date: 

Witness's Signature: Date: 
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Dear Formative Committee Members, 

Thank you for participating in my current research project. Your guidance is a cornerstone of this 
project. As a group, you will determine the criteria for success. 

Listed below are scenario type questions to help me determine the criteria for the interview instrument. 
Please answer the questions fully and honestly. I am looking for what you each feel will result in a 
successful interview instrument, not what you think I want to hear. You may consider this a brainstorming 
session, and no input will be rejected. Please do not feel like you must write a great deal of information 
(think quality over quantity). All input will be reviewed by myself, compared with current literature, and 
consolidated into a final list of criteria for the committee's approval. 

An important point to understand is that I am concerned with your input on the best way to design and 
conduct the interview instrument as well as the appropriateness of the specific questions. 

Some questions to help you focus, based on current literature, include but are not limited to: 

1 . Do the interview questions relate to the desired information? 

2. How much time will the interview take? 

3. How is the interview presented to the respondents? 

4. Is the interview and analysis free from bias? 

Please respond via email at your earliest convenience. I would like to submit a final list of criteria to you 
by 30 Oct 00. 

Scenario Questions; 

1 . (Faculty Point of View) You are a current faculty member in undergraduate education and may have 
ideas what is important for successful learning. If you were asked to participate in an interview to share 
your knowledge, what would you expect from the faculty interviewer and instrument? 

2. (Student point of view) You are an undergraduate student, with no training in teaching students. If 
you were asked to participate in an interview session, with a faculty member researching student 
evaluations, what would you expect from the interviewer and instrument? 

3. (Administrator point of view) You are a department head, and you require a faculty member from your 
department to conduct interviews on creating student evaluations of teaching. What would you expect in 
relation to the interviewer and instrument? 

Thank you for your time and effort. Please contact me with any questions you may have. This is still a 
committee and information may be freely exchanged. 

V/R 

Kevin C. Sellers 
KEVIN C. SELLERS, Capt, USAF 
Assistant Professor, Aerospace Studies 
Education Officer 
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Second Contact with Formative Committee 
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Dear Formative Committee Members, 

Thank you for your effort in helping me determine the criteria for a successful interview instrument. Based 
on your input, and my review of related literature, I have designated the following criteria, which the instrument 
must meet for success. 

Interview Instrument Criteria 

The interview instrument must meet the following criteria for success, which are broken into the areas of 
design and conduct: 

Design 

1. The interview instrument is designed for use in a face-to-face setting between interviewer and respondent 

2. The interview instrument is designed to obtain information pertaining to the needs of students and faculty in the 
area of teaching effectiveness 

3. The interview questions must not contain language that may bias responses 
Conduct 

1 . The interviewer creates a relaxed environment where the respondent may provide answers to the questions in 
confidence 

2. The interview is provided at the respondent’s convenience in consideration of time and location 

3. The interview is conducted in a similar manner each time with probing questions added only to clarify 
responses 

Now you each have an opportunity to review the above criteria, please respond in the following manner: 

1. Approved with no changes 

2. Approved with the following minor changes (list changes) 

3. Not Approved, needs major changes (list needed corrections) 

If the committee approves the criteria, I will create the instrument based on the above criteria. I will then 
submit the instrument to the formative committee for evaluation. After that evaluation, I will adjourn the formative 
committee. 

If the committee does not approve the above criteria, I will make appropriate changes and resubmit them to 
the formative committee. 

Please review the above carefiilly, as your input will determine the success of the interview instrument. If 
possible, please try to respond with your input by 7 November 2000 or earlier. If you have specific concerns, please 
contact me. 

Once again, thank you for your continuing effort. 

V/R 

Kevin C. Sellers 
KEVIN C. SELLERS, Capt, USAF 
Assistant Professor, Aerospace Studies 
Education Officer 
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The interview instrument must meet the following criteria for success: 

DESIGN 

1 . The interview instrument is designed for use in a face-to-face setting between interviewer 
and respondent 

2. The interview instrument is designed to obtain information pertaining to the needs of 
students and faculty in the area of teaching effectiveness 

3. The interview questions must not contain language that may bias responses 

CONDUCT 

1. The interviewer creates a relaxed environment where the respondent may provide answers to 
the questions in confidence 

2. The interview is provided at the respondent’s convenience in consideration of time and 
location 

3. The interview is conducted in a similar manner each time with probing questions added only 
to clarify responses 
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7 November 2000 

MEMORANDUM FOR FORMATIVE COMMITTEE MEMBER 
FROM: AFROTC DET/EO 



SUBJECT : Review of Interview Instrument Draft 

1. First, thank you for your assistance to this point; I appreciate the offer of your time and expertise. 
The final stage of the formative committee is at hand. It is now time for you to review the draft of the 
interview instrument. I have attached a copy of the instrument along with a feedback sheet. Please 
review the instrument, record your responses on the feedback sheet, and return the feedback sheet to me 
as an email attachment. 

2. To assist you in your review I have included the criteria you approved. They are the basis on which 
you should review the interview instrument. 

3. The interview instrument must meet the following criteria for success: 

Design 

• The interview instrument is designed for use in a face-to-face setting between interviewer and 
respondent 

• The interview instrument is designed to obtain information pertaining to the needs of students and 
faculty in the area of teaching effectiveness 

• The interview questions must not contain language that may bias responses 

Conduct 

• The interviewer creates a relaxed environment where the respondent may provide answers to the 
questions in confidence 

• The interview is provided at the respondent’s convenience in consideration of time and location 

• The interview is conducted in a similar maimer each time with probing questions added only to clarify 
responses 

4. Once again, thank you for your help and guidance. Please return the feedback sheet to me by 13 Nov 
00, if possible. If you have further questions contact me. 



//SIGNED// 

KEVIN C. SELLERS, Capt, USAF 
Assistant Professor, Aerospace Studies 
Education Officer 

Attachments: 

1 . Interview Instrument 

2. Feedback Sheet 
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Appendix I 
Interview Instrument 



Interview Protocol: Conduct the interview at the convenience of the respondent. When 
interviewing faculty members, conduct the session in their office, or in the department’s 
conference room. When interviewing students, conduct the session in the department’s 
conference room. Conduct the interview in a similar manner during each face-to-face interview 
session. Read the questions as written, and only pose further questions to clarify responses 
(never provide possible answers to respondents). Utilize hand-written notes as well as a tape 
recorder to tabulate responses. 

Personal Information 

1 . Is respondent a Student or Instructor? 

2. In which Academic Year is respondent involved? 

Interview Questions 

1 . What teaching techniques would make you conclude that an instructor is a good teacher? 

2. What teaching techniques would make you conclude that an instructor is a poor teacher? 

3. What teaching techniques would not alter your opinion of the instructor’s teaching ability? 

4. What aspects of the Air Force ROTC course, other than teaching technique, would make you 
conclude that the course was good? 

5. What aspects of the Air Force ROTC course, other than teaching technique, would make you 
conclude that the course was poor? 

6. What aspects of the Air Force ROTC course, other than teaching technique, would not alter 
your opinion of the course? 
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Committee Feedback Form 

Directions: Add your comments as necessary. I will make changes based on your input if 
needed. Upon completing the form, delete the two choices, under the recommendation section, 
that do not apply. Then, attach this document to an email and return it to me. 

Input on instrument’s effectiveness in meeting criteria: 



Input on the questions (type and wording): 



Input on the method of interview conduct: 



General comments: 



Recommendation (please retain choice and delete others): 



Approved (no changes) Approved (consider above changes) 



Not Approved 
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1 6 November 00 

MEMORANDUM FOR SUMMATIVE COMMITTEE MEMBER 
FROM: AFROTCDET/EO 

SUBJECT: Summative Review of Interview Instrument 

1 . First, thank you for participating in my research project; I appreciate the offer of your time and 
expertise. The final stage of the project is at hand. It is now time for the summative committee to review 
the final draft of the interview instrument. I have attached a copy of the instrument along with a feedback 
sheet. Please review the instrument, record your responses on the feedback sheet, and return the feedback 
sheet to me as an email attachment. 

2. To assist you in your review I have included the criteria approved by the formative committee. They 
are the basis on which you should review and validate the interview instrument. The design of the 
program should meet the stated criteria. Please make any comments about the instrument on the feedback 
sheet. 

3. The interview instrument must meet the following criteria for success: 

Design 

• The interview instrument is designed for use in a face-to-face setting between interviewer and 
respondent 

• The interview instrument is designed to obtain information pertaining to the needs of students and 
faculty in the area of teaching effectiveness 

• The interview questions must not contain language that may bias responses 

Conduct 

• The interviewer creates a relaxed environment where the respondent may provide answers to the 
questions in confidence 

• The interview is provided at the respondent’s convenience in consideration of time and location 

• The interview is conducted in a similar manner each time with probing questions added only to clarify 
responses 

4. Once again, thank you for your help and guidance. Please return the feedback sheet to me by 22 Nov 
00, if possible. If you have further questions contact me. 



//SIGNED// 

KEVIN C. SELLERS, Capt, USAF 
Assistant Professor, Aerospace Studies 
Education Officer 

Attachments: 

1 . Interview Instrument 

2. Feedback Sheet 
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Appendix L 
Tabulation of Results 



Question 1 


Instructor 


Students 


Freshman 


Communicates knowledge 
effectively^, enthusiasm , subject 
knowledge®, active student 
involvement® 


1 . Knows material , real-life experiences 
interested in subject, other students speak 
well of instructor"*, teaches in learning style 
similar to students®, reviews material®, 
helps students remember material®, 
accessible®, students comfortable with 
talking to instructor® 

2. Listens to students®, allows class input®, 
interactive®, gets students interested"*, 
visual aids^, relates to students®, 
understands what students need to learn® 


Sophomore 


Utilizes questioning®, discussions^, 
effective movement®, enthusiasm"*, 
allows class participation® 


1 . Class involvement®, does not read from 
book®, class participation and interaction®, 
attention step®, makes material easy to 
learn®, utilizes group projects®, reviews at 
end of class period® 

2. Various assignments®, work one-on-one 
with students®, gets to know students®, 
multiple examples®, enthusiastic"* 


Junior 


Not evaluated 

Researcher is junior instructor 


1 . Expresses difficult concepts well®, 
lectures but allows student interaction®, 
shares learning®, two-way 
teaching/leaming®, fim personality"* 

2. Knowledge of material*, real-life 
examples®, availability®, approachable®, 
encourages questions in class®, positive and 
upbeat atmosphere in class"*, interested and 
excited about material"*, concerned with 
students’ progress® 


Senior 


Clear communicator®, well 
prepared', organized', motivated"*, 
creates stimulating enviromnent®, 
uses a variety of support for 
objectives®, uses guest speakers®, 
uses external reference®, uses 
personal experiences®, balances 
need to present material through 
lecture and interactive methods® 


1 . Knows material*, does not solely follow 
book , uses language familiar to students , 
life examples®, handouts^, available to 
help®, encourages use of office hours®, 
makes students feel comfortable®, answers 
questions®, compassionate conununication 
skills®, reviews subject after each class® 

2. Uses discussion®, begins with lecture 
then discusses material®, uses relevant 
slides^, available®, works out problems in 
class® 
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Question 2 


Instructor 


Students 


Freshman 


Poor eye contact"^, turns back to 
students^, lacks knowledge of 
subject®, no organization 


1 . Not interested in material"*, lacks 
knowledge of material or requisite 
training®, too friendly with students®, 
teaches below knowledge level of 
students^, teaches to “dumbest” student^, 
treats students like children® 

2. Teaches at instructors knowledge level- 
not students^, monotone®, slow pace® 


Sophomore 


Reading book or lesson plan', poor 
eye contact^, no emotion"* 


1 . Lacks enthusiasm"*, just reads material', 
one-way communication®, lacks eye- 
contact®, does not know students names®, 
lacks accessibility® 

2. No motivation"*, boring"*, overuses 
examples®, talks down to students®, talks 
above students®, “babies” students®, talks to 
the board®, does not address questions®, 
inattentive to students® 


Junior 


Not evaluated 

Researcher is junior instructor 


1 . Verbal pauses (urns and ahs)®, does not 
care about students®, does not accept 
feedback from students® 

2. Lack knowledge of material®, must rely 
on notes or slides®, trouble conveying 
message®, does not vary teaching methods®, 
uses technical terms® 


Senior 


Difficulty communicating 
information^, difficulty relating to 
students^, not organized’, 
regimented presentation^, lacks 
humor, perspective, and animation"*, 
does not allow free-flow of ideas®, 
concerned over “ratings” more than 
presenting material"* 


1 . Only follows book', teaches too 
quickly®, monotone®, not enthusiastic"*, late 
for class’, does not know student names®, 
does not allow student questions®, 
embarrasses student who ask questions®, 
not available® 

2. No background material given to 
introduce topics®, forces students to 
respond to questions®, expects same 
enthusiasm for subject from students as 
teacher may have"*, pure lecture®, reads 
from notes®, no personal experiences 
presented® 



IISST COPY AVAIIABIjE 
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Question 3 


Instructor 


Students 


Freshman 


Dresses professionally^, 
timeliness*, utilizes audio visual 
methods'^ 


1 . Book or assignments required by 
department and beyond instructors control’ 

2. Methods of class preparation’, method 
of presentation^ 


Sophomore 


Utilizes audio visual methods**, 
excessive movement^ 


1. Ideals surrounding curriculum’, method 
of instruction^, cultural accent^, slang 
terms^ 

2. Visual aid quality^ 


Junior 


Not evaluated Researcher is junior 
instructor 


1 . Appearance^, race*^, gender*^ 

2. Appearance^, gender^ 


Senior 


Method of approaching lesson 
presentation', way class is 
organized (as long as it’s 
organized)’, choice of teaching 
method , personality traits (humor) 


1 . Only covers exam material*, does not 
present material m depth 

2. Using standardized slides^ 



Question 4 


Instructor 


Students 


Freshman 


Topics narrowed to objectives’, 
real-life scenarios^, mid-day 
timeframe^ 


1 . Students know material’’, good 
teaching , able to use material , interesting 
and relevant to students'*, various topics^ 

2. What students leam^, outside reading 
assigned^, various activities to apply 
material, various teaching methods 
applied^ 


Sophomore 


Organization’, fair quantity of 
material^, good test questions’ 


1 . Relaxed environment*’, student get along 
and help each other^, other students can be 
relied upon^ 

2. Material presented'*, how well students 
remember material^, content^, group 
presentations , hands-on , how well 
instructor teaches^, organized and efficient’ 


Junior 


Not evaluated 

Researcher is junior instructor 


1 . Point of view of instructor and text’, 
reflects various viewpoints’, material 
relates to future^ 

2. Material can be applied'^, relevant 
material'^ 


Senior 


Objectives are appropriate for 
students level of learning’, lesson 
plans are a good foundation’, 
enough time allotted to teach 
material^ 


1 . Interesting material'*, material has 
importance in life , keeps students awake , 
energetic students^, students left interested'* 

2. Material relates to future needs'’, 
material is specific in nature^ 
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Question 5 


Instructor 


Students 


Freshman 


Typos in text and handouts', poor 
room aesthetics^ 


1. Students do not like instructor ^ not 
designed around student learning styles', 
not taught to learning style of majority of 
students , early in morning or late at night , 
poor text', too much work* 

2. Not enough instruction to ensure 
understanding^, t3T?e and location of room^ 


Sophomore 


Poor textbook', lack of human 
interest^ 


1. Outside distractions^, student 
personalities brought into classroom^, 
boring or long subject matter"*, class length 
too long^, difficult location and time in 
relation to other courses^ 

2. Students cannot remember matenal , 
poor teaching technique^ 


Junior 


Not evaluated 

Researcher is junior instructor 


1. Teaching format^, students do not 
respect instructor^ 

2. Material not relevant*, unorganized', 
lacks plan or objectives' 


Senior 


Material not current', no 
technology utilized in class 
teaching (i.e. multimedia)^ 


1. Boring material"*, not informative*, 
opinionated material counter to student 
beliefs^ 

2. Book and class do not coimect'. Book 
material does not apply to lesson 
objectives', lacks real-life material*, lacks 
information relevant for future planning 



Question 6 


Instructor 


Students 


Freshman 


Temperature'*, room arrangement'* 


1 . Stated all things will register good or 
bad 

2. Time of class^, number of students^, day 
of week^, type of visual aid^ 


Sophomore 


Amount of material (if deletions 
can be made)* 


1. Substitute instructors', altering time of 
tests or quizzes', surprise speaker', uniform 
wear' 

2. Sizeofclass^ 


Junior 


Not evaluated 

Researcher is junior instructor 


1 . Race or gender of instructor'* 

2. Time of class^, what is covered (as long 
as material is relevant)* 


Senior 


Stated all things will register good 
or bad 


1 . Uninteresting required course"* 

2. History material included* 
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Appendix M 
Frequency of Results 

Note: Areas are numbered. The numbers appear as superscripts in Appendix L, thus providing 
information on what responses were counted in each area below. 

Category of Instructor Related Questions (questions 1-3) 



Areas Registering Concern (quest 


:ions 1-2) 


Area 


Frequency 


Average 


+/- From Average 


1 . Course Organization 


5 


18 


-13 


2. Effective Communication 


19 


18 


+1 


3. Group/Student Interaction 


17 


18 


-1 


4. Instructor Interest/Enthusiasm 


18 


18 


0 


5. Rapport/ Availability for Students 


27 


18 


+9 


6. Visual Aids 


3 


18 


-15 


7. Methods of Presentation/Leaming 


39 


18 


+21 


8. Instructor Knowledge 


8 


18 


-10 


9. Instructor Physical Characteristics 


0 


18 


-18 



Areas Registering No Concern (question 3) 


Area 


Frequency 


Average 


+/- From Average 


1 . Course Organization 


5 


18 


-13 


2. Effective Communication 


2 


18 


-16 


3. Group/Student Interaction 


0 


18 


-18 


4. Instructor Interest/Enthusiasm 


0 


18 


-18 


5. Rapport/ Availability for Students 


0 


18 


-18 


6. Visual Aids 


4 


18 


-14 


7. Methods of Presentation/Leaming 


6 


18 


-12 


8. Instmctor Knowledge 


0 


18 


-18 


9. Instmctor Physical Characteristics 


8 


18 


-10 
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Category of Course Related Questions (questions 4-6) 



Areas Registering Concern (quesf 


dons 4-5) 


Area 


Frequency 


Average 


+/- From Average 


1 . Course Organization 


17 


10 


+7 


2. Teaching Skill 


13 


10 


+3 


3. Course Value 


13 


10 


+3 


4. Interesting Material/Information 


7 


10 


-3 


5. Breadth of Coverage 


5 


10 


-5 


6. Learning Environment 


9 


10 


-1 


7. Other Students 


7 


10 


-3 


8. Workload 


2 


10 


-8 


9. Instructor Physical Characteristics 


0 


10 


-10 



Areas Registering Concern (question 6) 


Area 


Frequency 


Average 


+/- From Average 


1. Course Organization 


4 


10 


-6 


2. Teaching Skill 


0 


10 


-10 


3. Course Value 


0 


10 


-10 


4. Interesting Material/Information 


1 


10 


-9 


5. Breadth of Coverage 


2 


10 


-8 


6. Learning Environment 


8 


10 


-2 


7. Other Students 


0 


10 


-10 


8. Workload 


1 


10 


-9 


9. Instructor Physical Characteristics 


1 


10 


-9 
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